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discussed. 


IMPROVING  ASSESSMENTS  FOR  STRATEGIC  DECISION-MAKING  IN  COUNTER 

INSURGENCY  OPERATIONS 


Assessment  is  a  process  that  measures  progress  of  the  joint  force  toward 
mission  accomplishment.  The  focus  is  on  measuring  progress  toward  the 
end  state  and  delivering  relevant  reliable  feedback  into  the  planning 
process  to  adjust  operations  during  execution. 

— JP  5-01 

“How  can  we  show  irrefutable  progress  in  Afghanistan?”  This  basic  question 
posed  by  GEN  Stanley  McCrystal  to  his  staff  revealed  limitations  in  the  process  used  by 
his  assessment  teams  to  address  strategic,  operational,  and  tactical  applications  across 
the  command.  While  Joint,  North  Atlantic  Treaty  Organization  (NATO),  and  Service 
specific  literature  define  how  and  why  assessments  are  conducted,  they  generally  fall 
short  in  addressing  the  data  collection  required  for  the  complex  counterinsurgency 
(COIN)  environment.  Analysis  for  COIN  operations  is  very  challenging,  due  in  part  to 
the  need  to  understand  perceptions  and  culture,  the  need  to  track  hundreds  or 
thousands  of  personalities,  the  local  nature  of  insurgencies,  and  the  tendency  of 
insurgencies  to  change  over  time.2  To  measure  progress  at  the  strategic  level,  there 
must  be  data  collected  not  just  at  the  strategic  level  but,  also  at  the  local,  tactical  level 
where  insurgents  are  most  active.  These  measures  help  planners  understand  the 
operational  environment  to  frame  the  problem  and  develop  operational  approaches. 

This  monograph  specifically  addresses  a  proposed  methodology  to  collect 
relevant  data  that  provides  a  more  robust  assessment  of  the  operational  environment. 
The  paper  is  organized  into  six  sections.  Following  this  short  introduction  is  a  section 
on  assessment  doctrine.  The  third  section  examines  two  counterinsurgencies,  namely 
Vietnam  and  Afghanistan,  to  determine  where  there  are  specific  shortfalls.  The  fourth 


section  contains  a  proposed  framework  for  data  collection  and  data  sharing  pertinent  to 
the  strategic,  operational,  and  tactical  environments.  Additional  techniques  specific  to 
counterinsurgencies  and  transition  are  provided  in  the  fifth  section.  The  final  section 
contains  the  conclusions  of  this  research. 

Doctrine 

Assessments  occur  from  the  highest  levels  of  strategic  planning  in  determining 
the  Combatant  Commander’s  mission  areas  to  the  evaluation  of  a  tactical  mission  in  a 
larger  campaign.  All  assessments  provide  information  to  the  command  for  input  in  the 
decision-making  process  and  start  during  the  planning  process.  Specifically,  one  of  the 
purposes  for  developing  an  operational  approach  is  to  develop  the  assessments  for  the 
campaign.  The  measures  developed  aid  in  answering  performance  and  effectiveness 
questions  for  the  commander  in  reaching  the  desired  endstate.  During  an  operation  or 
campaign,  the  assessment  process  is  a  continuous  operation  that  coincides  with  the 
decision-making  process.  Generally,  the  process  consists  of  three  steps: 

1 .  Monitoring  the  situation  to  collect  relevant  information. 

2.  Evaluating  progress  toward  attaining  end  state  conditions,  achieving 
objectives,  and  performing  tasks. 

3.  Recommending  or  directing  action  for  improvement.3 

Relevance  of  information  will  be  dependent  on  the  mission  or  campaign.  It  is  not  limited 
to  the  Commanders  Critical  Information  Requirements  (CCIR),  but  the  types  of 
information  necessary  to  provide  situational  understanding  or  in  comparison  to  the 
forecasted  situation  in  the  commander’s  intent.  Counterinsurgencies  are  multi¬ 
dimensional,  therefore  relevant  information  should  inform  the  Political,  Military, 
Economic,  Social,  Information,  and  Infrastructure  (PMESII)  domains.  Responsibility  for 


2 


collecting  the  information  is  contained  in  the  Reconnaissance  and  Surveillance  plan. 
Given  the  complex  nature  of  counterinsurgencies  over  time,  there  should  be  an 
expectation  that  the  measures  that  are  considered  relevant  may  also  change  as  the 
environment  changes. 

The  second  step  of  the  process  is  evaluating  progress  and  this  is  where  the 
analyst  plays  a  key  role.  The  evaluation  is  critical  in  providing  the  commander  an 
analysis  of  what  is  working,  not  working,  and  potential  insights  on  how  to  better 
accomplish  the  mission.  Measures  of  Effectiveness  (MOEs)  and  Measures  of 
Performance  (MOPs)  are  generally  used  to  determine  and  communicate  progress 
toward  achieving  objectives  and  attaining  end  state  conditions.  Measures  provide  the 
criteria  to  judge  progress  and  may  be  objective  or  subjective  in  nature.  In  general, 
MOPs  help  to  answer  the  question  “are  we  doing  things  right”  versus  MOEs  that  answer 
“are  we  doing  the  right  things?”4 

The  third  step  is  to  recommend  or  direct  action  based  on  the  assessments.  This 
process  requires  the  entire  staff  to  evaluate  the  progress  of  the  campaign,  offer 
improvements,  and  make  an  estimate  of  the  merit  of  their  proposed  changes.  It  is 
important  that  the  staff  recommendations  are  integrated  by  the  chief  of  staff  and 
synthesized  to  provide  the  commander  a  vetted  recommendation  from  multiple  sources. 

The  assessment  process  follows  the  same  steps  for  a  conventional  or  irregular 
warfare  campaign  plan,  however  the  measures  vary  considerably.  For  a  conventional 
campaign,  measures  may  include  friendly  and  enemy  casualties,  orders  of  battle,  and 
terrain  held.  All  are  key  measures  for  determining  whether  a  conventional  campaign  is 
going  well,  and  can  be  used  to  inform  discussions  of  strategic  adjustment.5  Irregular 
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warfare  campaigns,  specifically  counterinsurgencies  are  centered  on  the  perception  of 
the  populace  which  is  much  more  difficult  to  measure.  The  next  section  contains  the 
analysis  of  two  counterinsurgencies  in  Vietnam  and  Afghanistan. 

Case  Studies 

This  section  will  examine  two  counterinsurgency  campaigns  in  light  of 
contemporary  doctrine.  The  first  campaign  examined  is  the  Vietnam  campaign  in  1966 
and  the  second  is  the  Afghanistan  campaign  in  2009-2010.  These  campaigns  are 
categorized  as  insurgencies  because  of  the  inherent  weakness  of  insurgent  forces 
relative  to  the  state  or  external  forces.  This  relative  weakness  forces  insurgents  to  avoid 
a  direct  confrontation  and  instead  look  for  ways  to  attack  asymmetrically.  While 
combatants  generally  prefer  a  quick,  cheap,  overwhelming  victory  over  a  protracted 
struggle,  insurgents  often  must  prolong  their  effort  to  gain  in  relative  strength  and  erode 
the  will  of  opponents  over  time.6 

The  operational  environment  for  Vietnam,  similar  to  the  contemporary 
environment  of  Afghanistan,  can  be  best  described  as  diverse  and  complicated. 

Several  consecutive  American  administrations  did  not  fully  understand  the  multi-faceted 
operational  environment  and  the  nuances  within  the  PMESII  domains.  From  the 
political  standpoint,  the  United  States  government  committed  early  on  to  Ngo  Dinh  Diem 
as  the  premier.  Diem’s  nationalism  and  administrative  experience  made  him  a  logical 
choice  for  the  premiership  of  an  independent  Vietnam,  but  he  lacked  many  of  the 
qualities  required  for  the  imposing  challenges  he  faced.7  The  government  he  inherited 
did  not  have  the  capacity  to  meet  the  needs  of  the  populace.  The  political  agencies  and 
services  still  in  existence  could  not  extend  out  to  the  rural  areas  that  were  often 
controlled  by  the  insurgents  who  lived  and  hid  amongst  the  populace. 
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The  French  departure  from  Vietnam  left  the  Vietnamese  military  poorly  equipped 
and  trained  to  meet  the  country’s  needs.  Initially,  the  United  States  advisers 
transformed  the  250,000  man  army  into  a  1 50,000  Soldier  force  in  mobile  divisions. 
During  this  process,  the  Vietnamese  force  became  dependent  on  the  American  aid  to 
keep  the  force  viable.  The  transformation  to  the  mobile  divisions  did  not  necessarily 
take  into  account  the  overall  rugged  terrain  of  Vietnam  and  poor  interior  road  network. 
Overlooking  the  infrastructure  capacity  of  the  road  networks  across  the  three  main 
sections  of  Vietnam  led  to  a  less  than  optimal  force  structure. 

The  Vietnamese  economy  subsisted  predominantly  on  the  export  of  rubber  and 
tin  to  meet  the  demand  for  imports.  The  economic  reform  and  nation-building  through 
development  projects  had  a  positive  effect  on  the  cities  of  South  Vietnam  but  did  little  to 
impact  the  majority  of  the  population  in  the  rural  areas.  This  further  exaggerated  the 
social  and  political  dilemmas  for  government. 

From  the  social  standpoint,  the  Vietnamese  people  comprised  various  religions, 
ethnicity,  and  occupations.  The  population’s  values,  beliefs,  and  interests  were  further 
divided  between  the  urban  and  rural  population.  Vietnam  did  not  consist  of  a 
homogeneous  population  where  one  approach  could  be  successful  for  the  nation  as 
Diem  attempted.  Diem  was  intolerant  of  the  social  demands  of  the  various  minorities,  or 
sects,  which  hampered  American  nation  building  and  allowed  the  seeds  of  insurgency 
to  grow  in  the  fertile  environment  of  the  rural  populace.  Diem  was  never  able  to  build  a 
strong,  legitimate  government  as  his  premiership  ended  with  his  untimely  death  in  the 
coup  of  1963.  “The  population’s  religious,  ethnic,  and  geographical  diversity  posed 
significant  challenges  for  building  a  popular,  self-sustaining  government.”8 
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Without  a  full  understanding,  or  at  least  an  appreciation,  of  the  operational 
environment,  the  United  States  developed  a  campaign  plan  for  the  Vietnam  War.  The 
campaign  plan  consisted  of  four  components: 

1 .  Air  Campaign  against  North  Vietnam 

2.  Nation  building  within  South  Vietnam 

3.  Diplomacy 

4.  Ground  battle  in  South  Vietnam 

Current  doctrine  defines  these  components  as  lines  of  effort,  yet  these  lines  of 
effort  do  not  relate  to  the  overall  problem  that  should  have  been  developed  from  an 
analysis  of  the  operational  environment.  Each  of  these  lines  of  effort  require 
assessment  to  gauge  the  progress  and  if  reframing  is  required  as  part  of  the  recursive 
process. 

Current  doctrine  describes  assessments  as  a  process  that  measures  progress  of 
the  joint  force  toward  mission  accomplishment.9  If  the  command  was  able  to  assess 
the  progress  along  these  lines  of  effort,  they  could  have  reframed  the  problem  to  match 
the  situation  on  the  ground.  Instead,  statistics  and  measurements  instituted  by 
Secretary  of  Defense  McNamara  took  an  unusually  prominent  role. 

Secretary  of  Defense  Robert  McNamara  was  a  key  figure  in  the  Vietnam  War 
and  the  assessments  process.  His  background  in  business  and  systems  analysis  led  to 
his  direct  involvement  in  the  measures  and  statistics  collected.  Unfortunately,  while 
there  was  a  great  appetite  for  data  from  Vietnam,  the  data  collected  did  not  support  the 
measurement  of  the  strategic  objectives.  Military  Assistance  Command,  Vietnam 
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(MACV)  Directive  88  issued  in  1963,  consisted  of  1 10  metrics  which  were  not  linked  to 
any  of  the  strategic  objectives.10 

By  1966,  the  strategy  in  Vietnam  was  modified  to  include  assistance  and 
governance  projects  in  an  effort  to  connect  with  the  populace  and  win  over  their  “hearts 
and  minds.”  In  this  case,  there  were  problems  with  the  objectives  that  made  it 
impossible  to  measure  progress.  For  example,  one  of  the  objectives  was  to  increase 
the  population  that  lived  in  secure  areas  by  ten  percent.  There  was  not  a  clear 
definition  of  how  this  should  be  measured;  and  there  was  ambiguity  in  what  the  overall 
objective  covered.  Was  the  intent  to  secure  more  area  or  did  it  mean  that  we  would  be 
successful  if  we  moved  the  populace  from  unsecure  areas  to  secure  areas?  There 
were  several  other  similar  objectives  that  were  so  ill  defined  as  to  be  useless  in 
campaign  assessments. 

One  of  the  objectives  that  was  measurable  entailed  increasing  the  destruction  of 
the  enemy  force  by  235,000.  The  measure  was  nicknamed  the  body  count  metric 
because  this  was  one  metric  that  was  actually  measurable.  However,  many  of  the 
enemy  “bodies”  were  never  recovered  that  were  reported  as  enemy  casualties.  An 
unintended  second  order  effect  of  this  measure  included  units  that  would  artificially 
inflate  their  enemy  body  counts  to  show  that  they  were  successful. 

As  a  means  to  measure  the  hearts  and  minds,  a  system  was  put  in  place  that 
would  survey  the  populace  to  gauge  their  perceptions  on  the  war  effort.  This  system 
was  called  the  Hamlet  Evaluation  System  (HES)  and  surveyed  the  rural  population  on 
items  such  as  medical  care,  village  infrastructure,  and  insurgent  activity.  RAND 
conducted  a  statistical  analysis  of  the  data  in  1968  and  found  the  data  was  statistically 


7 


stable  and  a  good  indicator  of  rural  development  programs.11  The  downside  to  the 
HES  was  the  enormous  amount  of  data  it  produced.  There  were  roughly  1 2,000 
villages  in  the  program  and  approximately  2,000  villages  were  surveyed  each  month.  A 
typical  report,  in  total  consisted  of  90,000  pages  of  data.12  While  an  analyst  could 
manage  a  few  villages,  no  human  could  possibly  do  a  temporal  analysis  on  the  country 
wide  set  of  data;  there  was  simply  too  much  data. 

From  the  Vietnam  case  study  there  are  three  specific  lessons  that  impact  our 
doctrine  concerning  assessments.  The  first  is  the  necessity  to  link  the  measures  to  the 
strategic  objectives,  which  implies  that  the  objectives  are  well  defined  and  measurable. 
The  second  is  that  data  collection  without  analysis  is  not  useful.  There  was  such  a 
demand  for  the  countless  metrics  in  the  Pentagon  that  it  seemed  more  important  to  get 
the  reports  on  time,  vice  the  quality  of  what  was  in  the  reports.  This  leads  to  the  third 
point,  that  too  many  statistics  could  not  be  integrated  into  an  operational  or  strategic 
construct.  It  was  not  due  to  a  lack  of  data,  but  the  sheer  magnitude  of  information  such 
as  from  the  HES  reports  that  could  not  be  combined  for  a  useful  assessment. 

The  second  counterinsurgency  case  study  examined  was  taken  from  Afghanistan 
in  the  2009-2010  timeframe.  At  the  end  of  2009,  the  Headquarters  (HQ),  International 
Security  Assistance  Force  (ISAF)  created  an  operational  command  at  the  three  star 
level  called  the  ISAF  Joint  Command  (IJC)  to  command  and  control  the  operational  and 
tactical  war.  HQ,  ISAF  would  remain  the  theater  command  under  NATO  and  Central 
Command  (CENTCOM)  reporting  chains  with  the  responsibility  for  the  strategic 
campaign  plan. 
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The  ISAF  strategic  campaign  plan  changed  over  time  to  reflect  the  environment 
both  internal  and  external  to  Afghanistan.  The  plan  consists  of  Lines  of  Effort  (LOEs) 
focused  on  security,  governance,  development,  and  neutralization  of  malign  networks. 
Each  LOE  has  subordinate  measures  that  consist  of  subjective  and  objective  measures 
with  indicators  collected  at  the  district  level.  The  data  is  aggregated  at  the  Regional 
Command  (RC)  level  and  forwarded  to  the  IJC  for  the  operational  assessment  on  a  six 
week  recurring  schedule.  ISAF  strategic  assessments,  informed  by  the  IJC 
assessments,  are  reported  monthly,  quarterly,  and  annually. 

Assessment  on  each  of  the  LOEs  consists  of  multiple  data  points  via  various 
reporting  mechanisms.  For  example,  the  Security  LOE  assessment  at  the  district  level 
consists  of  the  Significant  Activities  (SIGACT)  reports,  the  IJC  security  rating  for  the 
district  which  includes  the  commander’s  subjective  rating,  and  public  perception  data 
from  polling.  SIGACTs  consist  of  mandatory  reporting  events  such  as  enemy  contact  or 
improvised  explosive  device  (IED)  explosions  that  are  input  into  the  Combined 
Information  Data  Network  Exchange  (CIDNE)  database.  (CIDNE  is  the  authoritative 
database  for  the  Afghanistan  theater  and  contains  both  subjective  and  objective  data.) 
All  these  data  points  are  considered  in  the  overall  subjective  assessment  which  consists 
of  a  five  color  scale  rating  for  the  district. 

The  Security  LOE  has  the  most  objective  data  particularly  from  the  SIGACT 
reports,  but  there  is  a  lack  of  objective  data  in  the  Governance  and  Development  LOEs 
which  feed  into  the  political  and  economic  domains  of  the  PMESII  analysis.  The 
assessment  for  the  LOEs  predominantly  consists  of  the  subjective  Provincial 
Reconstruction  Team  (PRT)  narrative  reports,  IJC  ratings,  and  polling  results.  Specific 
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to  the  governance  and  development  sections  are  the  narrative  reports  developed  by  the 
respective  staff  sections.  These  estimates  are  developed  in  concert  with  the  United 
States  Embassy  and  Afghanistan  Ministries  where  appropriate.  These  reports  along 
with  hundreds  of  narrative  reports  are  extremely  difficult  to  objectively  assess.  Recent 
improvements  in  the  CIDNE  database  provide  for  the  inclusion  of  these  narrative 
reports,  but  these  reports  are  not  necessarily  quantifiable  or  repeatable.  For  example,  a 
Human  Terrain  System  (HTS)  report  on  economic  development  in  Marjeh  district  may 
be  only  one  data  point  that  is  never  repeated.  The  assessment  teams  have  the  difficult 
task  of  searching  for  the  governance  and  development  LOE  narrative  data  to  form  to  an 
overall  subjective  assessment  on  the  five  color  scale  rating  for  the  district. 

The  Afghanistan  assessment  process  is  much  closer  to  the  contemporary  Joint 
doctrine  and  follows  the  intent  to  link  measures  to  the  objectives.  However,  there  are 
several  lessons  to  learn  from  this  case  study.  The  first  is  to  have  a  relevant  data 
collection  plan  to  provide  discreet,  verifiable  data  to  drive  the  assessments.  Next,  if  a 
system  is  to  be  used  that  combines  qualitative  and  quantitative  measures,  then  this 
system  should  be  codified  such  that  it  is  replicable  and  reliable  over  time.  Finally,  there 
is  a  tremendous  amount  of  data  available  from  the  Afghanistan  Ministries,  Non- 
Governmental  Organizations  (NGOs),  and  international  organizations  that  could  provide 
additional  insight  to  the  assessment  process.  This  virtually  untapped  resource  could 
confirm  or  deny  trends,  and  if  found  reliable  could  reduce  the  data  collection  burden  for 
ISAF. 

In  comparing  the  Afghanistan  case  study  to  the  Vietnam  case  study,  there  are 
similarities  and  clear  distinctions.  The  case  studies  are  similar  in  that  they  both  have  a 
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tremendous  amount  of  subjective  data  collected.  In  the  Vietnam  case  study,  it  is  clear 
that  these  measures  were  not  linked  to  the  campaign  objectives  whereas  in  the 
Afghanistan  study  there  are  normally  weak  relationships  using  subjective  data.  While 
the  tools  to  maintain  the  data  have  improved  since  Vietnam,  analysts  in  Afghanistan  still 
struggle  with  the  amount  of  unstructured  narrative  data  to  assess  the  environment 
effectively.13 

Understanding  the  operational  environment  is  one  of  the  key  principles  in  the 
design  process  of  campaign  planning.  In  both  scenarios,  using  polling  data  in  Vietnam 
from  the  HES  or  using  various  national  polls  in  Afghanistan  fails  to  capture  the  relevant 
data  to  use  the  PMESII  construct  to  inform  the  commander  of  progress.  The  mis¬ 
understanding  of  the  operational  environment  can  lead  to  an  uninformed  problem 
statement  and  the  risk  of  defining  an  operational  approach  that  does  not  lead  to 
success.  Defining  a  structured  set  of  data  focused  on  the  operational  environment  will 
greatly  enhance  the  commander’s  situational  awareness  and  lead  to  more  informed 
decisions. 

Proposed  Framework  for  Data  Collection  and  Sharing 

This  section  describes  a  proposed  method  to  frame  the  data  collection  and 
sharing  problem  to  not  only  provide  the  commander  with  enhanced  situational 
awareness  of  the  operational  environment,  but  provide  a  means  to  show  progress  in 
campaign  assessments  with  sufficient  analytic  rigor.  The  proposal  begins  with  a  review 
of  the  foundational  data  available  in  Afghanistan,  categories  of  data,  and  the 
requirements  to  prepare  a  general  data  collection  plan.  Next  is  a  description  of  the 
framework  specifically  applied  to  the  Afghanistan  theater.  The  section  concludes  with  a 
discussion  on  the  challenges  and  lessons  learned  from  this  approach. 


11 


Foundational  Data.  There  is  a  tremendous  amount  of  data  available  concerning 
Afghanistan  and  sources  of  data  are  as  varied  as  the  amount  of  information  available. 
Identifying  appropriate  data  sources  to  support  assessments  in  Afghanistan  is  of  critical 
importance  to  operational  commanders  and  ISAF  member  nations.  In  general  the  data 
can  be  divided  into  two  categories,  namely  structured  and  unstructured  data.  The 
preponderance  of  data  in  Afghanistan  falls  into  the  unstructured  category.  That  is,  the 
data  is  of  a  qualitative  nature  consisting  of  a  textual  narrative  and  perhaps  a  subjective 
rating. 

Data  is  collected  across  theater  for  many  purposes  from  the  local  tactical 
commander  to  the  operational  and  strategic  level.  Examples  include  surveys  done  at 
the  local  and  national  level,  host  nation  information  requirement  reporting,  and 
significant  activities  reported  in  CIDNE.  Given  the  nature  of  a  counter  insurgency 
environment,  many  of  the  tactical  level  data  and  measures  have  strategic  implications. 
Unfortunately,  outside  of  CIDNE  there  are  very  few  means  for  sharing  information  in  a 
structured  database.  Further,  there  are  data  repositories  only  known  to  the  local  unit 
(i.e.  stored  on  unit  hard  drives)  and  therefore  are  not  shared  across  theater.  One 
shortfall  to  this  data  is  that  there  is  no  mechanism  in  place  to  even  identify  these 
sources. 

Data  collection  and  data  sharing  plans  must  be  synchronized  to  ensure  that  the 
proper  data  is  collected  at  the  right  place,  time,  with  the  appropriate  method,  and  then 
shared  over  acceptable  media  or  network  to  organizations  that  can  benefit  from  this 
information. 
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Methodology.  Every  assessment  starts  with  a  question  or  a  set  of  questions  to 
provide  an  indication  or  measure  of  progress.  Data  collection  and  sharing  of  this  data 
are  integral  to  the  success  of  the  process.  The  following  figure  is  an  example  of  how 
assessments  (or  similar  analyses)  are  conducted.  In  each  of  the  boxes,  there  is  an 
internal  process  that  develops  the  data  for  that  particular  section.  Each  of  the 
processes,  shown  in  Figure  1 ,  are  sequential  with  outputs  from  one  process  leading  to 
inputs  for  the  next  process. 


Issue  or  Question 

Collection  Plan 

Storage  and  Sharing 

-  Campaign  Assessment 

>=> 

-  Organization 

0 

-  Database 

-  Trend  Analysis 

-  Frequency 

-  Wiki 

-  CCIR 

-  Method 

-  Spreadsheet 

-  Qualitative,  quantitative 

-  Classification 

Did  we  ask  the 
right  question? 


Do  we  have  the 
right  data? 


Analysis 

-  MOEs,  MOPs 

-  Time  Series 

-  Exploratory  Data 
Analysis 


Figure  1 :  Data  Collection  and  Data  Sharing  Process  for  Assessments14 


The  process  starts  with  a  question  which  could  be  in  the  form  of  a  commander’s 
Information  requirement  or  a  specific  measure  related  to  Provincial  transition.  The  type 
of  question  will  determine  the  processes  involved  to  collect  the  data  necessary  to 
provide  the  commander  the  necessary  information  to  make  the  decision. 

One  of  the  first  processes  is  the  data  collection  plan.  The  result  of  the  data 
collection  plan  is  a  listing  of  the  measures  and  indicators  required  and  the  specifications 
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for  the  collection.  Specific  requirements  should  include  how  often  the  data  needs  to  be 
collected,  who  is  responsible  for  collecting  the  data,  and  what  method  of  collection 
should  be  used.  For  example,  survey  methodologies  are  quite  specific  on  how  many 
surveys  need  to  be  completed  at  each  level  for  the  results  to  be  valid  for  district, 
province,  or  national  level.  Other  factors  include  the  classification  level  of  the  data,  as 
well  as,  determining  if  the  data  should  be  quantitative,  qualitative,  or  both.  Generally, 
analysts  will  want  to  conduct  time  series  analysis  or  predictive  analysis  over  a  range  of 
data  points.  To  understand  the  trends,  data  will  need  to  be  collected  in  a 
methodological  fashion  at  specific  time  frequencies  such  that  the  analysis  is  valid. 

Once  the  data  is  collected  it  needs  to  be  shared,  or  at  the  minimum,  retrievable 
by  the  analyst  team  providing  the  assessment.  Therefore,  the  means  of  providing  the 
information  via  the  network  needs  to  be  considered  and  included  in  the  collection  plan. 
Military  forces  have  the  capability  to  pass  a  great  deal  of  classified  information  over  their 
tactical  networks.  The  same  capability  does  not  reside  with  government  and  non¬ 
government  agencies  trying  to  pass  information  over  non-secure  networks. 

Similar  to  sharing  the  information  via  a  network  transport,  storage  of  the  data 
must  also  be  considered.  Depending  on  the  complexity  of  the  data  and  intended  use, 
the  solution  for  storage  could  range  from  a  website,  to  a  SharePoint,  to  an  integrated 
relational  database.  Where  the  data  is  stored  and  who  can  retrieve  the  data  for  analysis 
needs  is  important  to  the  overall  quality  and  repeatability  of  the  assessment. 

The  process  does  not  end  with  the  analysis  cycle;  it  is  an  iterative  process  that 
continues  to  refine  the  questions  and  collection  process.  The  goal  is  to  provide  the 
commander  the  information  necessary  to  make  informed  decisions.  There  is  as  much 
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“art”  as  there  is  “science”  in  selecting  the  right  measures  and  collecting  data  for  them. 
Therefore,  there  is  a  need  to  incorporate  flexibility  into  the  data  collection  plan  linking 
various  sources  of  data  if  possible.  The  best  analysis  for  the  purpose  of  assessments  is 
time-series,  geospatially  linked,  quantitative  data  complemented  by  qualitative  data  that 
adds  context. 

One  of  the  main  considerations  in  the  data  collection  process  is  the  organization 
that  is  going  to  collect  the  data.  While  tactically  it  is  quite  simple  to  order  a  military  unit 
to  conduct  reconnaissance  and  surveillance  of  an  area,  there  are  additional 
considerations  and  alternatives  that  deserve  attention.  For  example,  if  valid  and 
trustworthy  information  can  be  garnered  from  other  sources  then  the  command  could 
use  this  source  instead  of  tasking  military  units.  This  provides  the  commander 
additional  flexibility  in  using  his  forces  for  security  or  other  missions. 

As  discussed  previously  in  this  section,  the  issue  being  examined  drives  the 
process  for  collecting  data.  In  the  case  of  ISAF,  there  are  specific  data  collection  assets 
in  place  to  collect  information  to  inform  assessments.  This  is  particularly  true  in  the  key 
terrain  and  area  of  interest  districts,  where  military  units  and  PRTs  actively  collect 
information.  In  other  areas,  sometimes  called  “white  space,”  there  are  many  disparate 
data  collection  efforts  that  include  ISAF,  Government  of  the  Islamic  Republic  of 
Afghanistan  (GIRoA),  as  well  as  NGO  and  other  organizations.  MG  Flynn  in  his  white 
paper  on  “Fixing  Intelligence”  refers  to  these  sources  as  rich  unclassified  data  sources 
that  provide  context  on  the  population.  Understanding  the  human,  sociological,  cultural, 
and  behavioral  (HSCB)  aspects  of  the  population  and  the  data  collection  efforts  that 
support  this  evaluation  are  critical  to  the  success  of  the  operation.15 
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There  are  many  data  sources  available  to  the  ISAF  for  assessments  from  military 


sources,  official  Afghan  government  sources,  intergovernmental  organizations  (IGO) 
and  non-governmental  organizations  (NGO).  The  NATO  authoritative  source  for  ISAF 
operations  is  the  CIDNE  database,  which  resides  on  a  classified  network.  In  order  to 
provide  unclassified  information  to  partner  agencies,  the  International  Distributed 
Uniform  Reporting  Environment  (INDURE)  provides  similar  types  of  information  as 
CIDNE  at  the  unclassified  level.  Many  of  the  Afghanistan  ministries  collect  data  for  their 
specific  charter  that  are  authoritative  and  applicable  for  use  in  assessments. 


Source 

Description 

Reports 

PMESII  Domains 

CIDNE 

Classified  authoritative 
database  for  ISAF 
operations. 

SIGACTs,  Intel, 
Targeting, 
Psychological 
Operations,  HSCB 

Primarily  Military, 
but  contains  data  on 
all  domains. 

INDURE 

Password  protected 
unclassified  version  of 
CIDNE  providing  a 
means  for  outside 
agencies  to  upload 
data. 

SIGACTs  and  other 
declassified  CIDNE 
reports,  civil  capacity, 
socio-cultural. 

Information 
collected  for  all  the 
domains. 

Central 

Statistics  Office 

GiROA’s  only  official 
statistical  body  and 
authoritative  data 
source  for  country 
statistics. 

Population, 

education,  health 

development, 

agriculture 

development, 

inflation,  exchange 

rate. 

Information 
collected  for  all 
domains,  but  limited 
on  Military. 

Ministry  of 
Rural 

Rehabilitation 

and 

Development 

Unclassified  repository 
of  final  reports  and 
data  on  projects 
across  Afghanistan. 

Roads,  education 
projects,  civil 
projects. 

Mainly  Economic 
and  Infrastructure. 

Table  1 :  Small  Sample  of  Military  and  Government  Databases. 
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Two  specific  examples  include  the  Central  Statistics  Office  (CSO)  and  the  Ministry  for 
Rural  Rehabilitation  and  Development  (MRRD).  A  description  of  the  government 
sponsored  data  sources  are  listed  in  Table  1 .  Additional  sources  concerning  NGOs  and 
other  media  are  listed  in  Table  2. 


Source 

Description 

Reports 

PMESII  Domains 

Afghan-lnfo 

Comprehensive  listing  of 
hundreds  of  websites  on 
Afghanistan.16 

Media,  news, 

government, 

political, 

reconstruction, 

economy. 

All  domains. 

World  Bank 

Provides  statistics 
concerning  international 
development 
measures.17 

Education,  poverty, 
medical, 

employment,  etc... 

All  domains  except 
for  Military. 

NGOs 

Many  organizations 
provide  information  on 
their  activities  in 
Afghanistan  via  the 
Internet  or  published 
reports. 

Various  depending 
on  the  location  of 
the  projects  and 
type  of  work. 

All  domains  except 
for  Military. 

Academia 

Universities,  institutes, 
and  think  tanks  publish 
research  relevant  to 
understanding  the 
operational  environment. 

Various  depending 
on  the  research. 

All  domains. 

Traditional 

Media 

Regular  reporting  of 
events  that  can  be 
searched  or  data-mined 
through  a  Nexus  search. 

Articles  that  report 
on  events  and 
public  opinion. 

All  domains. 

Non-Traditional 

Media 

Sources  consist  of 
blogs,  twitter  feeds,  and 
internet  only  journals 
that  usually  have  less 
quality  control  than 
traditional  media. 

Various  reports  that 
target  a  specific 
audience. 

All  domains. 

Table  2:  Sample  of  Additional  Data  Sources. 
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With  all  the  data  sources  that  are  available,  the  analyst  team  needs  to  determine 
the  validity  and  accuracy  of  the  data  for  its  intended  purpose  in  the  assessment.  Data 
sources  could  be  scored  to  evaluate  the  differences  between  the  sources.  Suggested 
categories  include,  but  are  not  limited  to  the  following:  collection  method,  credibility  of 
the  source,  accessibility,  relevance,  sustainability,  format,  and  confidence.  Data 
sources  that  directly  inform  a  specific  measure  should  be  the  highest  scoring  but  that 
does  not  mean  that  a  low  scoring  data  source  should  be  discarded.  The  low  scoring 
data  sources  may  provide  valuable  contextual  information  or  support  a  minority  opinion 
on  some  assessments.  One  effort  called  Datacards,  sponsored  by  the  Office  of  the 
Secretary  of  Defense  (OSD),  attempts  to  catalog  and  score  the  accuracy  of  the  sources. 
Datacards  is  a  structured  collection  tool  that  indexes  data  sources  that  relate  to  irregular 
warfare,  assessment,  or  can  be  used  for  socio-cultural  modeling.  These  cards  provide  a 
summary  description  and  evaluation  of  the  content,  quality,  intended  purposes,  and 
potentially  appropriate  uses  of  each  source.18 

The  data  landscape  in  a  conflict  environment  will  always  be  challenging.  Some 
of  the  main  challenges  include  discipline  in  collection,  over  classification  of  data,  lack  of 
incentives,  and  validation.  With  respect  to  discipline  in  data  collection,  there  is  a  need 
to  continually  evaluate  data  collected  to  ensure  it  is  answering  the  question.  Leadership 
needs  to  enforce  the  accurate  collection  and  timely  upload  onto  the  appropriate 
network.  There  is  a  balance  between  overburdening  the  unit  with  too  many 
requirements  to  collect  data  and  collecting  the  most  important  data.  When  units  are 
overburdened;  data  collection  will  lapse,  errors  will  be  more  pronounced,  and  reports 
will  be  delayed  or  not  reported  at  all.  One  method  to  alleviate  some  of  the  burden  in 


18 


reporting  is  the  use  of  standardized  templates.  Alternatively,  the  staff  can  find 
information  that  was  collected  for  another  purpose  (proxy  data)  that  can  provide  an 
acceptable  answer. 

There  is  a  tendency  to  over  classify  data  in  a  conflict  environment. 
Counterinsurgency  operations  in  particular  will  generate  information  related  to  the 
population  that  is  generally  unclassified.  This  information  is  sometimes  referred  as 
“white  data”  and  can  provide  context  to  the  current  situation  and  provide  useful  proxy 
measures  in  security,  governance,  and  development.19  Collecting  and  sharing 
unclassified  information  provides  a  larger  audience  of  analysts,  particularly  in  academia, 
to  work  on  current  problems. 

Data  incentives  are  extremely  important  to  the  units  that  collect  data.  On  both 
the  military  and  civilian  side  of  operations,  the  data  should  be  useful  to  the  organization 
collecting  the  data.  For  instance,  units  provide  IED  related  data  because  they 
understand  that  the  reports,  trend  analysis,  and  overall  situational  awareness  will  be 
improved  if  they  do  so.  They  see  tangible  products  that  affect  their  operations  which  is 
a  huge  incentive.  Not  all  incentives  will  be  this  pronounced,  but  we  cannot  expect  an 
NGO  or  other  organizations  to  provide  data  to  a  system  that  doesn’t  provide  them  utility. 

Finally,  data  validation  will  continue  to  be  a  challenge.  Analysts  will  not  always 
be  able  to  use  authoritative  data  sources.  For  example,  surveys  are  one  source  of 
information  used  in  measuring  the  perception  of  the  populace.  There  are  various 
methodologies  and  sampling  techniques  used  in  survey  design  and  analysis  which  must 
be  understood  to  begin  the  validation  process.  Analysts  should  also  look  for  correlation 
between  data  sources  as  one  of  the  measures  in  validation.  These  challenges  are  not 
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insurmountable  and  analysts  at  all  levels  have  been  providing  incredibly  detailed  and 
accurate  assessments  to  their  commanders. 

Emerging  Techniques 

In  an  effort  to  better  understand  the  operational  environment  in  Afghanistan, 
several  efforts  have  been  suggested  in  varying  degree  to  provide  additional  information 
to  the  commander.  One  such  technique  is  through  Host  Nation  Information 
Requirements  (HNIR)  and  Specific  Information  Requirements  (SIR).  These  measures 
were  developed  by  members  of  the  IJC’s  Information  Dominance  Center  to  capture  the 
complex,  nuanced,  and  dynamic  environment  of  Afghanistan  across  multiple  domains. 
These  measures  should  provide  a  much  clearer  picture  than  polling  data,  anecdotal 
references  and  statistics  insufficient  for  true  understanding  within  the  partnered 
commands.20 

HNIRs  essentially  capture  information  on  the  population  to  build  the  critical 
operational  environment  necessary  in  operational  design.  The  HNIRs  can  be  collected, 
shared,  analyzed,  and  disseminated  similar  to  CCIRs  using  the  data  collection  methods 
previously  discussed.  These  potentially  disparate  bits  of  information  need  to  be 
synthesized  across  domain  experts  to  provide  the  context  for  the  commander  to  better 
understand  the  whole  environment  and  Host  Nation  capability  from  the  lowest  level 
possible.  The  quantitative  data  combined  with  qualitative  expert  opinion  across  the 
civilian  and  military  communities  provides  a  mosaic  of  information  across  all  levels  of 
command  and  leadership.21 

SIRs  are  information  requirements  for  an  area  of  operations  linked  to  an 
organization  that  can  collect  the  information.  The  unique  feature  of  SIRs  is  that  the 
organization  collecting  the  information  could  be  tasked,  but  just  as  often  would  provide 
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the  information  in  a  cooperative  arrangement  without  a  tasking.  For  example,  an  HTS 
team  could  be  asked  to  gather  information  on  a  bazaar  in  their  area  on  a  recurring 
basis.  The  HTS  team  falls  under  their  Brigade  Combat  Team  chain-of-command  where 
it  is  assigned  and  is  not  under  the  direct  tasking  authority  of  the  IJC.  Another  example 
includes  asking  an  NGO  or  IGO  team  to  report  on  a  bazaar  in  a  location  where  there  is 
not  an  ISAF  presence.  In  the  latter  case,  the  NGO  or  IGO  team  would  need  incentive  to 
report  the  information  which  could  be  assuaged  by  providing  them  access  to  an 
unclassified  data  source  that  would  be  relevant  to  their  needs. 

The  process  to  capture  the  information  is  similar  to  the  process  shown  in  Figure 
1  with  a  notable  exception.  This  particular  process  calls  for  a  multi-disciplinary  team  to 
synthesize  the  information  in  a  common  analysis  center,  a  shareable  information 
system  across  the  military/civilian/partner  stakeholders,  and  the  analysis  disseminated 
to  applicable  staff  sections  for  assessments.  “It  is  about  maneuvering  information  to  the 
commander... achieving  more  effective  partnership  through  shared  unity  of 
understanding.” 22 

Dr.  David  Kilcullen,  a  noted  expert  on  counterinsurgency  operations,  also 
developed  new  techniques  on  capturing  information  relevant  to  the  operational 
environment.  He  similarly  states  that  the  nature  of  the  information  in  the  population¬ 
centric  domain  does  not  readily  fit  into  the  stove  pipe  data  collection  process  to  answer 
questions  on  a  specific  LOE  and  argues  for  a  cross-domain  collection  process.  The 
majority  of  the  data  collected  falls  under  the  security  LOE  because  of  the  life  and  death 
consequences  that  coalition  forces  face  every  day.  However,  governance  and  security 
LOE  data  remains  just  as  important  for  the  long  term  success  of  the  campaign  plan. 
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Security  gains  without  similar  improvements  in  governance  can  cause  a  rift  in  the 
expectations  of  the  populace  that  the  insurgents  can  exploit.  He  further  states  that 
there  is  a  need  for  consistent  measurements  over  time  on  a  common  set  of  metrics  with 
observable  indicators.  Many  of  these  measures  can  provide  proxy  data  for  several 
LOEs  while  providing  quantitative  data  to  show  change  in  the  operational  environment.23 

Kilcullen  lists  over  30  specific  measures  that  are  useful  in  this  manner  that  are 
categorized  into  four  areas:  security,  governance,  host  nation  capabilities,  and  enemy. 
The  measures  generally  are  characterized  by  providing  information  on  multiple 
domains,  most  often  are  quantitative  in  nature,  and  focus  on  the  change  between 
reporting  periods.  For  example,  one  measure  examines  the  price  of  agriculture 
products  (not  grown  in  the  district)  as  a  proxy  measure  of  security  and  development. 
Development  from  the  standpoint  that  there  is  commerce  involved  which  includes  the 
import  or  transportation  of  the  products.  Because  transportation  is  involved,  there  is  an 
inherent  cost  factored  in  for  security.  Kilcullen  theorizes  that  if  there  is  a  large  increase 
in  the  price  of  imported  food  then  security  and  or  corruption  is  probably  poor  in  that 
district.  Other  measurement  examples  include  the  voluntary  reporting  of  lEDs  and  the 
number  of  local  officials  that  live  in  the  district.  In  the  case  of  persons  turning  in  lEDs, 
this  is  already  in  place  as  a  means  to  indicate  security  gains.24  Each  of  these  measures 
provides  information  indirectly  of  the  security  in  the  district,  but  more  importantly  on  the 
overall  confidence  in  the  populace  of  the  government  and  coalition  effort. 

Kilcullen’s  measures  provide  complementary  data  to  the  assessment  process 
that  are  measurable  and  relevant  across  LOEs.  The  data  collected  from  this  process 
does  not  remove  the  qualitative  expert  judgment  from  lower  level  tactical  commanders 
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but  provides  the  quantitative  evidence  of  change.  Further,  the  measures  can  assist  in 
correlating  the  data  to  why  there  are  changes  in  the  environment.  Visualization  of  this 
data  can  lead  to  greater  situational  understanding  by  the  commander  and  reframing  of 
the  operational  approach  as  necessary. 

Generally,  military  commanders  are  extremely  comfortable  with  information 
provided  on  a  map  and  can  absorb  the  information  more  readily.  Visualizing  information 
on  map  is  critical  in  providing  the  context  of  the  information  and  shared  understanding.25 
The  Training  and  Doctrine  Command  Analysis  Center  at  Monterey  has  recently 
completed  a  research  project  concerning  exploratory  data  analysis  (EDA)  to  visualize 
operational  data  for  current  operations  in  Afghanistan.  Their  research  examined  the 
factors  and  processes  to  implement  a  software  solution  to  visualizing  the  data.  They 
found  that  the  two  most  important  characteristics  of  the  data  are  the  temporal  and 
geospatial  components.  In  the  most  basic  sense,  time  series  or  trend  analyses  relies 
on  the  temporal  component  of  data  collected  over  time.  Typically,  these  measures  are 
binned  monthly  but  can  be  queried  to  fit  the  time  interval  required  for  the  analysis.  The 
geospatial  component  of  the  data  is  equally  important  in  the  overall  analysis.  The 
human  brain  can  detect  trends  in  data  from  visual  displays  much  easier  than  from 
numeric  tables.  Providing  data  that  is  tagged  with  location  information  enables  the 
analyst  to  provide  much  more  information  to  the  operational  commander  on  a  map 
background.  Figure  2  depicts  unclassified  SIGACT  reports  over  Regional  Command 
South  (RC-S)  and  Regional  Command  Southwest  (RC-SW).  This  one  chart  depicts 
instantly  where  the  insurgency  has  concentrated  their  activity  and  the  relative  frequency 
of  attacks.  This  is  the  type  of  information  that  can  readily  inform  the  commander  to 
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make  decisions  on  where  to  adjust  force  strength,  as  well  as  government  and 
development  projects.  Further,  since  this  information  is  unclassified,  it  can  be  shared 
with  NGO/IGO  partners  leading  to  their  risk  decisions  on  whether  to  conduct  projects  in 
specific  districts. 


Figure  2:  Exploratory  data  method  to  geographically  show  security  incidents.26 

Conclusion 

Counterinsurgency  operations  represent  one  of  the  most  nuanced,  complex,  and 
dynamic  types  of  warfare.  Victory  in  these  operations  hinges  on  the  perception  of  the 
populace  and  not  on  the  destruction  of  the  armed  forces.  In  the  Vietnam  case  study, 
the  primary  metric  used  to  gauge  success  was  the  “body  count”  measure,  which  was 
prone  to  error  and  did  not  provide  the  insight  necessary  to  gauge  the  populace’s 
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perception.  In  Afghanistan,  it  took  until  2009  for  the  intelligence  community  to  realize 
the  shortfall  of  information  and  the  lack  of  a  collection  process  to  garner  information  on 
the  population.  Both  of  the  case  studies  lead  to  the  undeniable  conclusion  that  the 
commander  needs  to  have  a  thorough  understanding  of  the  operational  environment  in 
order  to  design  the  operational  approach  for  the  campaign  plan. 

To  measure  progress  in  the  campaign  plan  requires  careful  selection  of  the 
objectives  and  the  process  to  measure  the  objectives.  The  measures  collected  or 
derived  from  reliable  sources  must  be  sensitive  to  changes  in  the  PMESII  domains  to 
provide  the  commander  insight  into  changes  in  the  operational  environment.  Specific  to 
the  counterinsurgency  environment  in  Afghanistan,  there  are  numerous  open  source 
databases  that  provide  useful  information  to  assess  the  factors  that  impact  the 
population’s  perspective.  As  assessments  are  important  at  all  levels  of  command,  a 
data  sharing  environment  needs  to  exist  with  appropriate  classification  levels. 
Specifically,  this  network  needs  to  have  a  robust  unclassified  portion  to  share 
information  with  civilian  organizations  and  a  means  to  get  civilian  organization’s  data 
into  the  database.  The  Datacards  effort  under  the  Office  of  the  Secretary  of  Defense 
illuminates  a  portion  of  the  databases  that  are  available. 

Finally,  the  data  needs  to  be  synthesized  and  communicated  to  the  commander 
with  clear  and  convincing  evidence  on  the  progress  of  the  campaign  plan.  Using 
graphs  that  help  the  commander  visualize  the  information  on  a  map,  such  as 
exploratory  data  analysis  techniques,  aid  in  the  rapid  understanding  of  the  environment. 
Commander’s  at  the  strategic  level  need  accurate  information  on  the  environment  to 
make  the  most  informed  decision.  Without  this  analysis,  commanders  are  forced  to 
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make  decisions  based  on  instinct  which  carries  a  tremendous  amount  of  risk  in  a 
volatile,  uncertain,  complex,  and  ambiguous  environment. 
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